YOUR EVALUATION PLAN SHOULD INCLUDE THE FOLLOWING COMPONENTS: Intervention population Intervention target - Indicators Target outcomes (Intermediate/Ultimate Indicators) Referent group Confounding factors

 

As discussed in Step 1- Population

This is the population that your program has targeted as a priority or at the very least, the one it intends to reach. In your evaluation, you have to be able to show that the intervention population has indeed been reached.

 

As discussed in Step 3 - Targets. These indicators can help us evaluate whether each of our interventions was successfully implemented.

 

3. Target Outcomes (Intermediate/Ultimate outcomes)

These are the targeted measures that the intervention is designed to impact. Although the term includes the word 'outcomes,' these measures may or may not be caused by the intervention itself. Indeed, a basic requirement of any evaluation is to use them in the baseline period, to compare with the post period in the intervention population only, to the pre-period in the external referent (if available). Also, define the level of your intervention related to how to measure it (e.g. if you collect information about referrals to a Pulmonary Rehabilitation program, you should also collect information on whether this intervention was received or not by the patient, second level).

 

One of the ways to show that your implementation strategies have been effective is to present the results that you have achieved with your intervention as compared to a referent group that has not received this intervention.

Finding a referent is very important:

If you are keeping track of the total number of COPD admissions in your hospital, and one year, you notice increments in your numbers, you cannot necessarily attribute the increase in admissions to the failure of your interventions, or to more patients deciding to be treated in your hospital (you are treating them better), or to the population in your area having increased or gotten older (your hospital is an "open" environment). 

• Broadly defined, referents include control groups, expert opinion, the defined time period for the 'pre-intervention' period in a pre/post-intervention study (involving either the same individuals or a different group of people), historical control groups from a time period prior to that of the intervention population, and the concurrent period to the intervention population.

Evaluation Methods in Disease Management Studies: 2004-07 [Expert Opinion] -

The referent can be chosen by an experimental method (e.g. randomization) or a non-experimental method (e.g. people who opt out, staggered roll-out, an employer group that did not participate in disease management, a matched control group, a previous period in time for a patient population).

Finding the proper referent is even more important:

In the same example, one approach would be to compare your institution to another hospital in the same area; there might be a number of limitations, such as getting the data from the other hospital, and you would still not be able to quantify the patient mobility between the two hospitals.

Another approach would be to create a caseload of known COPD patients (who have been admitted at least once to your hospital) and to keep track of the number of admissions over a given period of time before and after the intervention; there would still be the limitation of accounting for the evolution of the disease, which could be somewhat mitigated by choosing a relatively short time period.

These are factors, which may or may not be measured, that could be independently related to the outcome metric but are not part of the intervention.

For example:

At the same time that you are implementing your disease management program, a new COPD drug could become available in your hospital formulary; you may end up with very good results in terms of the number of prescriptions for this treatment, but they may not necessarily be related to your intervention alone.

Table 3: Example of an evaluation plan

	Target intervention	Outcomes
		Intermediate (Care delivery and practice-oriented)	Ultimate (Patient and use of healthcare services-oriented)
Intervention population: Patients admitted to the hospital	Implement a discharge planning (D/C) tool   Indicator: % of patients admitted for COPD for which a D/C planning tool has been completed	Refer patient to COPD community services Indicator: % of admissions with a referral to the community clinic	Reduce re-admission rates   Indicator: In a given caseload, % of re-admission rates for COPD at 3-month and at 1-year intervals
Referent group	Because this is a new intervention, we can only compare subsequent years after implementation	a) Previous years (by chart review) b) Patients for whom a D/C planning tool was not completed	Previous years for same caseload (from database)

Evaluation is the systematic gathering, analysis and reporting of data about an intervention/
program/pathway to assist in decision-making.